First, you need to install the “Java Runtime Environment (JRE) 5.0 Update XXX” from sun, make sure you get the “Windows x64”-Version from the page (It doesn't matter if you download the AMD64-version, it will work on intel)
After you've done this, open ImageJ.cfg (in ImageJ-folder) with your favourite text editor and change it to:
C:\PROGRA~2\ImageJ\ C:\Program Files\Java\jre1.5.0_17\bin\javaw.exe -Xmx5000m -cp C:\PROGRA~2\ImageJ\ij.jar ij.ImageJ
Make shure that you add the correct paths (pay attention to “Program Files (x86)” in 8.3-Notation and stuff like that. This makes ImageJ use the 64bit-Version of Java (the 32bit-Version can still be left installed…). -Xmx5000m makes ImageJ use 5GB of RAM, so this should be more than enough.
(taken from ImageJDocu)